Goto

Collaborating Authors

 pr 0




Supplementary material for the paper The emergence of clusters in self attention dynamics

Neural Information Processing Systems

This appendix is organized as follows: Appendix A: Well-posedness results. Throughout the remainder of the paper, we use the terminology "tokens" Definition 3 (Equi-compactly supported curves) . To prove Proposition A.1, we show a more general result concerning global existence and uniqueness We will make use of the following lemma regarding ( 6). R. We now show ( 8). R), which finally leads us to ( 9).


Agnostic Reinforcement Learning with Low-Rank MDPs and Rich Observations

Neural Information Processing Systems

There have been many recent advances on provably efficient Reinforcement Learning (RL) in problems with rich observation spaces. However, all these works share a strong realizability assumption about the optimal value function of the true MDP . Such realizability assumptions are often too strong to hold in practice. In this work, we consider the more realistic setting of agnostic RL with rich observation spaces and a fixed class of policies ฮ  that may not contain any near-optimal policy. We provide an algorithm for this setting whose error is bounded in terms of the rank d of the underlying MDP .


Multi-dataset and Transfer Learning Using Gene Expression Knowledge Graphs

arXiv.org Artificial Intelligence

Gene expression datasets offer insights into gene regulation mechanisms, biochemical pathways, and cellular functions. Additionally, comparing gene expression profiles between disease and control patients can deepen the understanding of disease pathology. Therefore, machine learning has been used to process gene expression data, with patient diagnosis emerging as one of the most popular applications. Although gene expression data can provide valuable insights, challenges arise because the number of patients in expression datasets is usually limited, and the data from different datasets with different gene expressions cannot be easily combined. This work proposes a novel methodology to address these challenges by integrating multiple gene expression datasets and domain-specific knowledge using knowledge graphs, a unique tool for biomedical data integration. Then, vector representations are produced using knowledge graph embedding techniques, which are used as inputs for a graph neural network and a multi-layer perceptron. We evaluate the efficacy of our methodology in three settings: single-dataset learning, multi-dataset learning, and transfer learning. The experimental results show that combining gene expression datasets and domain-specific knowledge improves patient diagnosis in all three settings.


Data-Driven Approximation of Binary-State Network Reliability Function: Algorithm Selection and Reliability Thresholds for Large-Scale Systems

arXiv.org Machine Learning

While exact reliability computation for binarystate networks is NP-hard/#P-hard, existing approximation methods face critical tradeoffs between accuracy, scalability, and data efficiency. This study evaluates 20 machine learning methods across three reliability regimes--full range (0.0-1.0), high reliability (0.9-1.0), and ultra-high reliability (0.99-1.0)--to address these gaps. We demonstrate that large-scale networks with arc reliability 0.9 exhibit near-unity system reliability, enabling computational simplifications. Further, we establish a datasetscale-driven paradigm for algorithm selection: Artificial Neural Networks (ANN) excel with limited data (size < m), while Polynomial Regression (PR) achieves superior accuracy in data-rich environments (size m). Our findings reveal ANN's Test-MSE of 7.24E 05 at 30,000 samples and PR's optimal performance (5.61E 05) at 40,000 samples, outperforming traditional Monte Carlo simulations. These insights provide actionable guidelines for balancing accuracy, interpretability, and computational efficiency in reliability engineering, with implications for infrastructure resilience and system optimization. Keywords: Binary-State Networks; Network Reliability Approximated Function; Reliability Thresholds; Dataset Scalability; Artificial Neural Networks (ANN); Polynomial Regression; Monte Carlo Simulation (MCS); Binary-Addition-Tree Algorithm (BAT); BAT-MCS 1. INTRODUCTION Modern infrastructure systems--from power grids and communication networks to IoT ecosystems--demand rigorous reliability analysis to ensure operational resilience. These systems are often modeled as binary-state networks, where components (arcs/nodes) operate in either functional (1) or failed (0) states [1, 2, 3]. Within this paradigm, network reliability--the probability of maintaining 2 connectivity between specified nodes under given conditions--serves as a critical performance metric [4, 5-7].


Representation and Regression Problems in Neural Networks: Relaxation, Generalization, and Numerics

arXiv.org Artificial Intelligence

In this work, we address three non-convex optimization problems associated with the training of shallow neural networks (NNs) for exact and approximate representation, as well as for regression tasks. Through a mean-field approach, we convexify these problems and, applying a representer theorem, prove the absence of relaxation gaps. We establish generalization bounds for the resulting NN solutions, assessing their predictive performance on test datasets and, analyzing the impact of key hyperparameters on these bounds, propose optimal choices. On the computational side, we examine the discretization of the convexified problems and derive convergence rates. For low-dimensional datasets, these discretized problems are efficiently solvable using the simplex method. For high-dimensional datasets, we propose a sparsification algorithm that, combined with gradient descent for over-parameterized shallow NNs, yields effective solutions to the primal problems.


Statistical Inference in Classification of High-dimensional Gaussian Mixture

arXiv.org Machine Learning

We consider the classification problem of a high-dimensional mixture of two Gaussians with general covariance matrices. Using the replica method from statistical physics, we investigate the asymptotic behavior of a general class of regularized convex classifiers in the high-dimensional limit, where both the sample size $n$ and the dimension $p$ approach infinity while their ratio $\alpha=n/p$ remains fixed. Our focus is on the generalization error and variable selection properties of the estimators. Specifically, based on the distributional limit of the classifier, we construct a de-biased estimator to perform variable selection through an appropriate hypothesis testing procedure. Using $L_1$-regularized logistic regression as an example, we conducted extensive computational experiments to confirm that our analytical findings are consistent with numerical simulations in finite-sized systems. We also explore the influence of the covariance structure on the performance of the de-biased estimator.


Temporal Shift -- Multi-Objective Loss Function for Improved Anomaly Fall Detection

arXiv.org Artificial Intelligence

Falls are a major cause of injuries and deaths among older adults worldwide. Accurate fall detection can help reduce potential injuries and additional health complications. Different types of video modalities can be used in a home setting to detect falls, including RGB, Infrared, and Thermal cameras. Anomaly detection frameworks using autoencoders and their variants can be used for fall detection due to the data imbalance that arises from the rarity and diversity of falls. However, the use of reconstruction error in autoencoders can limit the application of networks' structures that propagate information. In this paper, we propose a new multi-objective loss function called Temporal Shift, which aims to predict both future and reconstructed frames within a window of sequential frames. The proposed loss function is evaluated on a semi-naturalistic fall detection dataset containing multiple camera modalities. The autoencoders were trained on normal activities of daily living (ADL) performed by older adults and tested on ADLs and falls performed by young adults. Temporal shift shows significant improvement to a baseline 3D Convolutional autoencoder, an attention U-Net CAE, and a multi-modal neural network. The greatest improvement was observed in an attention U-Net model improving by 0.20 AUC ROC for a single camera when compared to reconstruction alone. With significant improvement across different models, this approach has the potential to be widely adopted and improve anomaly detection capabilities in other settings besides fall detection.


On the Convergence of Coordinate Ascent Variational Inference

arXiv.org Artificial Intelligence

As a computational alternative to Markov chain Monte Carlo approaches, variational inference (VI) is becoming more and more popular for approximating intractable posterior distributions in large-scale Bayesian models due to its comparable efficacy and superior efficiency. Several recent works provide theoretical justifications of VI by proving its statistical optimality for parameter estimation under various settings; meanwhile, formal analysis on the algorithmic convergence aspects of VI is still largely lacking. In this paper, we consider the common coordinate ascent variational inference (CAVI) algorithm for implementing the mean-field (MF) VI towards optimizing a Kullback--Leibler divergence objective functional over the space of all factorized distributions. Focusing on the two-block case, we analyze the convergence of CAVI by leveraging the extensive toolbox from functional analysis and optimization. We provide general conditions for certifying global or local exponential convergence of CAVI. Specifically, a new notion of generalized correlation for characterizing the interaction between the constituting blocks in influencing the VI objective functional is introduced, which according to the theory, quantifies the algorithmic contraction rate of two-block CAVI. As illustrations, we apply the developed theory to a number of examples, and derive explicit problem-dependent upper bounds on the algorithmic contraction rate.